Using the Crowd to Do Natural Language Programming

Natural language programming has proven to be a very challenging task. We present a novel idea which suggests using crowdsourcing to do natural language programming. Our approach asks non-expert workers to provide input/output examples for a task defined in natural language form. We then use a Programming by Example system to induce the intended program from the input/output examples. Our early results are promising, encouraging further research in this area. 1 Natural Language Programming Deep understanding of natural language sentences is a challenging task. After around half a century work in this area, we don’t know of a single robust deep understanding system except in extremely narrow domains (Liang, Jordan, and Klein 2011). When it comes to describing programming tasks, deep understanding becomes even more challenging, as no vagueness/ambiguity could be tolerated. The challenge is so real that in a decade when there was a lot of optimism about the future of natural language understanding, Dijkstra (1979) called the idea of natural language programming “foolish”. Three decades later, with growth of programming forums, natural language programming has become very popular especially for the novice programmers. Hundreds of post are created every day describing one’s intended task and asking for a piece of code in a specific programming language accomplishing the task. Some companies such as Microsoft hire many programming experts to address the programming needs of the users of their products such as Microsoft Excel (Gulwani 2011), often defined in natural language form (possibly accompanied by one or more examples of the input/output). As a result, natural language programming is becoming a real issue, which needs to be addressed in one way or another. Following the successful deployment of crowdsourcing to replace/help the AI systems, returning answers in nearly real-time (Bigham et al. 2010), we are proposing a novel idea involving crowdsourcing to approach the problem of natural language programming while avoiding deep language understanding. Copyright c © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 2 Programming by Example In the core of our model lies a Programming by Example (PbE) engine. PbE (a.k.a. Programming by Demonstration or PbD) is one of the most studied topics in the realm of automatic programming (Cypher et al. 1993). The idea of PbE is to provide a system with many examples of input/output of the programming task, and try to induce a program only by looking by analyzing examples. For example, consider the following three input/output examples for a particular task.